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ABSTBiCT \ ' \ ^ 

This paper discusses tfae kinds ot vocabulary* itess 
retained by' a group of native Gerian* speakers in t>e context of ^n 
EnglishManguagie proficiency exasination. Its aain purpose is to. 
indicate the kind of inforaation that can he gathered using cllz'e 
procedures^* with the ain to integrate this inforaatXpn into further 
learning str^tegieis foj the acquisition and retention of second 
language voc^bu^Wy. students of JBi^gllsh at a Gersan university, vere 
given an exalplnatlon testing vocabulary/ reading coaprehension^ 
syntax^ and translation. & statistical analysis of cloze responses ^ 
was undertaken* Results rjaveal that not only are the correct 
responses , to cloze passages of interest, btit also those responses 
considered ^incorrect«'^ The auth^s investigates the l^inds of 
vocabulary ^teis generate^~in Response" to the cloze itess. In all but 
a .few cases, a high degree of structural sensitivity ^ Was exhibited by 
the students. Cloze iteas,' along vith whatever e^se they say reveal 
about t£e readability ^of a given pros.e text or about other language 
skills, cam^pro^ide inforaation about the vocabulary rajuge o^f a^ group 
of subjects. In -dealing With as ^dictionary of. iskpectancy,^ students 
reveal the strategies used in selecting a finite set of words froa a 
potentially infinite' second language #ocabulac7« 
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A considerable amount of research has been conducted on 
»the use' of tl^e cloze procedure in language learning and 
teaching. It is not the purpose of this pape^ to review all 
of it. John OlUr (1979) has dope this very well and should 
be referred to. Nor is,it^ the purpose of this paper to 
describe the^henomenon of cloze as a psychological ^ 
construct. There are a number of, articles on the psychology 
of cloze, of which the most basic ones-^^ described* in / 
James (197.9) as well as Oiler (1979). What will be / 
discussed is work that is being done to determine the kisdls 
of vocabulary items retained by a group' of native speaketp 
of German and revealed in the context of an English- language' 
proficiency examination. It is thus the' main purpose of 
this pap^r to indicate the kind of information that can,' He 
gathered using cloze procedures, with an eye to integrating 
the information in^o further learjiing strategies for tlfie 
acquisition and retention of second- lan^Age vocabulai 



* Research into the use of cloze, procedures as measu^:e% of 
vocabuliry^ retetition reveals that little has been done to' 
analyze the kinds of information that students .prov^^e in ^ 
-responding to a^^iven blank or set of blanks in a s^le of^ 
continuous* prose text. AborU, Rubenstein and Sterling 
(1959) describe the phenomenon^ of inverse proportionality of* 
grammatical class size to the predictability of a Jiven 
respons€f. The*fewel the, members, of a class (preposition, . 
conjtftiction, article); tlie fewer the possible- responses that 
will need to be analyzed,"^ that, is, the more possil:/i'e it is 
to predict accurately which items will be selected 
rlacGinitie (1961) criticized this conclusion becaJose it was 
based on "short" contexts, J^'e., those^in single sentences , 
not longer texts. At the same time fi^ 'discovere(J that - 
larger contexts, that is,' those withimore than f^ve words to 
the left or right, of a given blank, \did not nec^sarily help 
produce acceptable r^ponsJ^s, Colera^n and Millar (1967) 
reinforced MacGini|:ie*s conciusioijs^, sfiowing thit contexts 
beyond twenty words from a given cloze item do not 
contribute sigi^if icantly to its resolution, Th^y qualify 
this, however, by pointing out that the context/ must npt be 
mutilated, specifically, words iri the%:ontext fiust not, be' 
"clozed" themselv^. Oiler (1975) maintains! however, that 
larger context-^oo help resolve a cloze item, for so- 

called "function" words like prepositions an<i /articles , In 
addition, James (1979) points out that once a|cloze item^ has 
been resolved it contributes to the resolutioh of other 
items proactiveLy and retroactively. To what extent an 
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Vincorrectly" resolved blank contributes negatively to, the 
resolution of a blank further on in a text has not been . 
investigated, 'Practical experience shows, however, that 
students rarely return to*i;fems which they have not resolved 
the first time around' with information th^t they may hav^ 
gathered' from other ^iteros. This is no doubt due to the 
»"discrete-pbint*^ habits instilled' in students; . they may be 
.encouraged to make Mse of all 'the information giv^ in a 
reading text, but treat each test' item as a segar^te 
subtest, independent of other subtests/items .^^^ 
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First, the procedures us^d. All students of English at 
the Universitat Erlangen-Nurnberg- in Erlangen, Federal 
-Republic t>f Germany, are. reqilired to' part ici^te in ^ two- 
and-ra-half hour examination, held at the beg-in^iing of each 
acadcls^i.c semester. The particular version qf the' 
exafflinatioh to be xef>orte<l on*here was held on April 24,- 
1979;; and contained 120^tems, in tife fallowing four skill 
groups: y Vocabulairy (30 f ill-in-the-blanfc items); 2) 
Reading Comprehension (two "brief reading passages, one ' 
followed by 8 multiple-choice questions, the other Allowed 
by 7, AS well as one cloze passage ^ith^ 03 blanks , ^which is 
the siAject of this, paper); ^) Syntar-^iO" fill-in's calling 
_for insertion of the appropriate preposition^ 15 completion 
exercises with "scrambled" wofds as cues, 15 fill-in's 
calling for the insertion of the correct verb tense, and 5 
sentiences to be' rewritten in indire^^ct* speech); and 4) 
Translation (.15 German sentences to be translated intS' 
English). Thei examination just described serves as an. 
entrance examination for new students as \iell as -the final 
examination for students who have completed a oneisemester 
grammar/reading course. The items in the examinarip'n change 
from semester to "^emestet, so that one semester's entrance 
examine^ become ne;ctv semester ' s final examinee, with two' 
different examinations at t-|ie' beginning and at the end of a 
given semester. - Thus the ttwo groups to be, described here 
irepresent those sjtudents without any formal exposure to 
cloze procedures '^rior to the- examination and thosfe who in 
the course of the sem63ter prieceeding the t^^t had been 
/ given reading ^exts ^ith cloze items and wl/o were thus ^ 
' familiar w it h the procedure. * ' ^ % . 

Second, the reading text used. On the basis of tiie- 
previous experience the text was f^lt to be of a^. level of 
.difficulty commensurate with what the studentiS weir^ already 
familiar* although the actual text was unfamiliar to them. 
No attempt was made to mechanically blank every nth Aem^. 
Injone case, in fact, the distance betWe'en twb blanks was • 
onfy one word, ajid the indefinite article "a" et that! The 
blanks were selected on th^ fcasis of the structural features 
they represented, so ttiat a' variety of Vorm/f^inction typfes 
attained. Thus there were six- nouns,' ^hree verbs, one 
proposition of loc'ation, one phrasal verb (with 
prepositional complement), one conjunctitm^ one adjective, 

, and -one verbal (participial adjective). At this poipt it. 
should be* mentioned that no deliberate attempTt was mafde to 
select items purely on the basis of^lheir sentential'*.'* 

# .function, i.e. subject vs. objector subj ect -verb , agreement . 
No3>*^R^e items selected because of particular lejcical or 
refer^tial functions they might possess, although' the 
selections of one nouns over another, does reflect this 
interest. The .resulting. fifteen blanks thus represent. a 
vajriety of cont^nfe/ function tjrpesiy as. the following analysis 
will show. , . : 
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Thlrd,> the data. Although information is available on ^ 
.the overall performance of . both groups on other parts of the 
examination, battery , only the result^ on the cloze paift'will 
be /discussed here. The data on. each 'student across each 
cl^ze item is given in Tables 1, 2a, and 2b. Table 1 * . 

represents^ the beginning students (labeled EST = ^ 
"Einstufungstest"), of whom there were 29. Tables 2a and 2b 
^eprtsent those who students with already one semester of 
university-level ESOL instruction behind them, (labelled GKK 
= "Gr.undkursklausur") of whom there were 94. 

* » 

The students were ranked in these tables on the basis of 
the .number of "correct^' responses on all" 15 items, with the 
humber of verbatim (or exact)* responses having precedence in 
the ranking over- the number of verbatim plus acceptable 
responsi^s,* followed by thos^ students^ with acceptable plus 

* unacceptable responses, whereby an unacceptable I'esponse is 
one which was considered by the scorer to be unacceptable or 
which w^s simply not given, i.e., blank: -Thus' in Table 1 ' 
Student E6 ig -listed ahead of Student E7, because the fornjer, 
had 10. verbatim items and 1 acceptable, while the latter had 
9 verbatim and 2 acceptable Note, of course, that Student 
E14, with 8 verbatim and 4 acceptable (equals 12 total 

, correct) responses is listed* after Student E15, bec^Mse the 
latter had 8- verbatim responses, but only tw9 additj.onal 
correct ^acceptable) responses., giving a totdl of ^nly IC 
correct response^ overall. * 



On, the basis pf the mean scores on the cloze par*^ of the 
examination (15 items = '15 points), it was determined that' 
the mean scores- of the two groups w§re statistically \ 
similar, that is, not significantly different, using the 
-Student t distribution (t = .411)- Using the F "distribution 
on" the variances of the two groups yielded no statistically 
significant difference as well (F = 1.08). , This held tr\ie 
even when a sample of those students with previpus 
experience with cloze (the GKK group) was taker!, based on 
merely considering every third student out df the total 
group of 94, so that a subsample of 31 was available to 
compare with the students without cloze experience (EST n = 
29) . 



This would se^m *to indicate' that no special training is^ 
necessary for the population of subjects under 
investigation. At this point it 'should be statjed that all 
the students discussed here came from secondary schools" 
where English as. a foreign language is a mandatory subject 
for anywhere from five to nine years,, so that we* are dealing" 
with subjects with considerable exposure to a large number 
of vocabulary items, as wiell as to^ ^ variety of contexts, in 
English. ^ . ^ 

However, if 'one delvei into thte individual items, one 
nqticefe, divergences between the two groups, fn order to 
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compar^ one item with another, the percentage of Students 
^"'"'^^wering an item correctly, that is, either with a verbatim ^ ^ ^ ' ^ 

''or acceptable response, was computed for each item in 'each ^ 
^roup, The ^Percentages are given in Table 4. The items of . , . 

perhaps greatest interest* are tljpse Where the deviation 

between the two -groups is 10% or more, that is. Items iz, 3, ' - " 

9, and 15, although Items 8 and 10 could "^also be included, ^ 4 
with 8% and 9% discrepency respectively. Of the six items ^ 
just mentioned, only Items 3, and 9 show a highei percentage * ^ * 

y of beginners giving correct answers, on them than their mojre 

^experienced colleagues'. Since, of course, a change^^f one ' * 

or two subject-responups hks less impact on a'largfe Sample 
\ than on a small sample, a comparison' between two roughly 
} equdf groups w<^uld seem to^ be in order. The comparison 
" Isubsa^ple , with n=* 31, shows that <](|ly^in Item#3 was. the 
, \0% discrepency in pertormace" between the EST an3 the GKK' 

group maintained. . • . ^ * ' ' * ^ 

What, however, is of more^ interest is the kinds of . ' 

vocabulary items generated by the two groups* in .response to * " .* 

the cloze items. This means .that not only are the correet * ^. ^ . 

ijespoftses of intere^ but also those responses considered 

"incorrect." It^is at this point th^t 'formal statistical * . • 

« analysis becomes less useful, ^ince we are no longet • 
i^erest^d in, fot example, item discrimination or item 
^ difficulty. Rather 'we are interested ip. the actual lior^s , ^ • f 

perceived by the students^as fitting into the blank spaces^ ^ • 

From a linguistic point of view, therefore, it is valuable ' . 

to examine the. words actually ^iven as responses. * 

' ' \ * • . ' 

Before. investigating individuSt items, it should be ' . 
stated that in all but a few cases a high degree of • 
structural sensitivity vTas exhibited by both groups of * - ^ 

students. Thus, where a' noun -was expected, a noun wa^ 
produced. Where a preposition wa$ mandatory, a preposition 
was used, as in Item 14. If a pait tense was tie only one 
plausible, as in- Item 8, this was^j^ in fact, given. Even ' * 
^ certain semantic constraints were adhered to, such is that ' " \£ • 

in Item 7, which iif the. majority of cases produced word ^ 
related to. the Semantic algorithm "huroan-female-young- . * 

married." Where a noticeable divergence between what is ' — ~ 

expected and what is' produced exists^, such as in Item 14, 
i,%h^ incorrectly -cho^fen response "by" '(as opposed to "witii") . » 
lies within the realm of possibility fj^r colloo^tioJi with , ' 
the preceding vpyb, "entertain, " realistic . ^ 

paraphrase for the expression, „*'.*^^^l\^d to entert^ain the 
, company With pianoplay^jgj^::k/l2ts " f >. had to entertain the* , , 

.compafcay by playing -th'e'pi^no ..,." Curiously, this avoidance • ^ 

of "with" is contrary to normal German usag6,» since the 

equivalent ^f "with" <"mit") would l?e expjBcted. ' v , . , 

What strikes one almost immediately is the very large 
number of dlfftfrpnt re'sponses given to several items. One 
woul^ expect ^group^of 123 students t6 coro*e up with many 

^* Ok. % - . • • . 
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differeftt responses >ot an open-ended mesaure. Most of these 
responses should beN^sy to categbrize as acceptable or not 
acceptable, probable or improbable^, likely or unlikely. 
Howevej:, when the number of distinct responses approaches 
the total number of students in the sample group, the 
responses requii^e closer examination^ Questions of ^ 

^ acceBjtable or non acceptable begin to take on highly 
subjective overtones. For this purpose, the author devised 
two interrelated measures, one reflecting the number of 

^ distinct correct responses accepted, as compared with the * 
•total nuitJber of distinct answers produced, the other 
reflecting the total number of subjects responding to the 
item. Thus, with the first measi^re, which will be 
, designated "C," there is normally only one correct response - 
for all subjects, but ther.e could be up to n incorrect 
responses. In t^e case of cloze measurement, Jiowever, there 
mig^ be more 'than one -response which could be deemed 
correct, which could possibly render such a measure 
meanihgless. In practice, only t'hose responses will bet 
considered as potentially correct that are ^ctu^lly given. 
The decision to accept or reject a re^j^onse is admittedly 

^'highly subjective* but weil within the bounds normally 

' tolerated "by students and fellow teachers al^Jte for the 
acceptance or rejection'of an anamolous answer! This makes 
the second measure-,desi§;nated as "T," more .realistic , J^nce 
th^ actual number of different responses is taken into 
account, whether correct or incorrect, and then compared - 
with the*'iiumbef*bf students -subject participating in the 
measurement. ^ , . 

The two measures have the same mathematical limits, but 
* in reverse. . "C" has limits of +1 only tlK^redetermined 
.correct response is ^crfosen* by all subjects) to 1/n (each 
subject ^ives a different resi^^onse). "T" has the li^nits 1/n 
(only one distinct response is given) to 1 (each subject 
^ives a distinct response)^. A computational Example. Item 
7 yielded one verbatim and. three acceptable responses, or ' 
four correct responses , ^f or the beginning (EST) group. 
There we^e, including the cprrect ^responses , a total of nine 
distinct . response^ given, including zero responses. Thus" 
"C" for the ESf group is .44. For the nine distinct . ^ 
►responses there 29 subjects, {producing a VT" value of .31 ^ 
for the EST group. Across the page, the GKK group also had * 
-feb»3^flmft mnftber o£ correct responses, -4, aj^d. the same number 
of total respon^sV\9, for a "C" of .44' again. However, 
tihere were 9U^t thei4, yielding, a "T" value of .10. 

Most eyaluators feel that they have developed at*least a 
reliable measill?^4£. the greatest number of subjects chooses 
the- fewest nui^ber pf responses to a test item, that is, the ' 
majority of tiiem chooses only one response, the one 
designated "best" or "mostrcorr'ect Thus a "C" value of • 
1.0 might be considered optimum, yhile a "T" value 
approaching 0 mi^ht be considered best. What do theseHwo 
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•measures tell us? For test construction purposes, an item 
which yields the highfst*"'C" value and. the lowest '*T" value 
would seem to be best^for inclusibn in future te|ting," ' 

•taking into account, of course, the fact that cloze items 
can*not ^e' readily removed from their context without 
ceasing to be "c^loj^e items" in the traditional sense. On 
the-other hand, an iteflLiike #6 (".^. look for ...), which 
- ' has a "C" of .SS' and a "T" of .07; retains its usefulness as 
» '%Dhrasal verb test item in a number of contexts. It«m 7 is 
also useful as a t^st ftem, because of the semantic envelope'^ 

• around, it which makes it semantie«rt Ij- and culturally stable. * 

4p^ith this in mind, the' following items from the experimental 
text seem to be "good" items for other tests: 1, 4, 6, and 
7. On the other hand, the items with a low (= minus) "C" to 
"T" relationship seem likely candidates for a semantic 

, "psychogram' describing the kinds of Vocabulary elements 
present^ in th*e memories of a group of students. ^ In this 
respect the following items frOm the 15 in the cloze test, 
are interesting: 2, 5, a, 9, and 12. * Two of these items) 5 - 
and 12 J will now be discussed as to what »they seem to tell 
# • • us, about the students' perception of the task involved 

determining *the correct respqpse to the item. , 

With Item 5 not one of the EST group and only five of the 
94 students in the GKK group gave the- vefbatim response/ 
("l.. walking distance ...") to a stimulus ot* 4tfhid:h most . 
. most native speakers of Englieh would score almost 

perfectly, it being practically a fixed phrase in.English.' 
. - Howeveij^ very few o.f the G^rmdn native speajcers, in all 
their '5+ years of English in school, apparently ever*^ 
encou^ered this particular fttCution. And only one student 
' ^ in -either group gave a res^jons^ which was judged to be even 
% ^ partially acceptable, "... hiking distance ,"^ although 

,4, most native speakers would no doubt reject this as odd at 

.best. The other responses fall into two* rough groups, one 
^* w.ith -ing partici)>ial adjectives and the other with other - 
adject^Lves. -In the EST group there is a small number of 
students wh$ gave prepositions (for, in, of), whicff can be. 
, • explained by assuming that they thought that the preceeding 
word, "easy," ^ a noun, and that they were dealing with a 
sequence: noun/preposition/noun. Thefe^were almost as many 
st^udents who gave *a present participial as those who did not 
(8 to 14). Thclslkin the rorme^r group apparently reali^d 
that the word^olagKf^a^ more closely linlced to "distance" 
^ than to "easy," i^nd^^nrird^jref lect some kind of continuous 
^ action, to be matkedf by a veri>aJ with'mng. The others 
apparently did not nealize that, in this 'context an<J with 
*'^^ I • this ^orthographic rendering (there is, after all, n* comma 
• ^^^^^ "^asy"), that .another adjective parallel to "easy" 

4l (good, short, etc) is not expected. What many students 

gave was the word "reachable" or another word ending in 
-abl-e, such as "accessible," "approacJ^ble,'" "attainable,'' 
• or "passable." Vfhy? Because the closest equivalent in 
CJerman is "erreichbar ," the ending "-bar" in German being 
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morph^semantically equ&l to English "-able." Although the- 

German word for "distance" is* '-^Entf ernung" and a German 

would not normally collocdte "erreichbar" 'with "Entfernung," 

a slight semantic shift fVom "distance" to "goal" and from 

there to German "Ziel"CF goal) would yield a predictable 

English-German correspondence: "reachable (attainable?) 

goal" vs. "erreichbares, Ziel" despite the - incongruity of 

associating "goal" with "distance." , 

♦ 

• • • 

Interestingly enough, 27% of the GKK group did not 
attempt to resjjond to 'this item at all, as oppjased to only 
10% of the EST group. . This could mean that jthe- students who 
were more experienced in handling cloze texts thought it 
better to leave the Atein blank than to put in something that 
they knew could not be proper EnglisK. This did^ot, of 
course, stop, 15% of them, as well as 17%^of the beginners, 
^from following their German instincts and inserting 

reachable" where it doesn't belong. Item 12 shows a 
tighter perception of which response is sought. Both groups 
seem^ to know which verbs in English can normally be 
followed by "up:" grow,^ bring, divide, make, dress, draw, 
get, pin, run,, tear. For this reason, th^ responses which 
are considered unacceptable are unacceptable not because 
they cannot be structurally associated with what follows t^e 
blank, but rather are unacceptable beqause they do toot fit 
- the blank semanti ca41y^ A native s pealcer is m o s t^ Ixkel y t:o 
associate "up" with '"grown" ^n the context of , family 
members, regardless c^^ other immediate semantic signals, 
something the native speakers of German in this sample did 
not. Yet many did -know of the connection between "up- 
bringing" and "growing-up" so that the decision' to reject 
anything that was not exactly "grcwn-up" was difficult. Jn 
a subsequent administration of the text to a group of native 
speakers, a decision was 'made to accept expressions sucH as 

dressed-up" and. "dolled-up," since these two options were 
considered acceptable by, it is assumed, proficient native 
speakers, although the examiners had previously decided to 
reject these tWo options as un^keptable. In a subsequent 
exercise rented to the use of tffe English preposition cum 
verb complement the teacher would know at least "which words 
the stujients associate with it 'and 'take appropriate 
pedagogical action. J « ) 

The^^ reader will have noticed that the tWo measures "C" 
and "t" resemble a standard ►statistical mapsure, item, 
-difficulty. However, both taJce into accoiiht the fact that 
there are 'measurement instrimnts v^hich do not have a fixed 
"correct" response to l;he exclusion of all others, and that 
the nupber of actual responses, -correct and incorrect, yield 
• important i^iformation about noV only the group of subjects 
being examined but also about the ability of an examination 
instrument . to generate this information. Cloze iten^, 
whatever else they- may tell us about the readability of a 
given prose text or about other language "skills with which 
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they a£e frequently correlated, can provide ipformation 
about the vocabulary range of a group of subjects . In sense 
. we are dealing with ^ "dictionary of expectency" paralleling 
Oiler's "grammar bf expectency" (1974) wbich can help 
predict the possible choice of words wl>*ch students can , be* 
expected to draw from when deciding wKich word to insert y 
into a blank space w'ithiij the procedures defined as "cloze." 
Vfhat- is thus described are the strategies that students use . 
in selecting "a finite set of words from a potentially 
infinite second- language vocabulary. 

' ■ • / 
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"Family Reupi^s." 

Before ^heJ Industrial ■ (1) scattered populations and steam 
and l^ter. petrol engines ensured easy mobility, there could 
have' been few family retinions as we know thisr ceremony at 
the present time, , with its assembling of inf r^quen^tly- 
encountered but still familiar: figures and the^ consequent 
getting'up to date with th^. events in other people's (-2). 
There must 'have been indeed a good deal of "dropping^in," 
for if members of a (3) did not actually live (4) the 
sam^e house, |:hey were within easy (5) distance of one 
another. A mans rarely went very far to look' (6) a (7) , 
and the girl who married someone from elswhere almost 
certainly said 'a filial goodbye to her parents when she w^s 
taken off to her husbiand's home. , , ^ • 

* * • » ' 

. It was the Victorians who firmly (8) tHfe family ' V \ 
re\inion. Present-day impressions of this imporjfant (^7"^^ , J 

arej5f a strange i>len(ling of formality', plain speaking withy . ' 
hovering dissension and a general (10) of natural warmth. 
(11) rarer such meetings were, it would seem, the more 
important it was to reflect prosperity and propriety.* Firm 
attention was pai4 tb suitable clothes and decorum while 
hostesses must ^demonstrate their ability to provide'- abundant 
meals- in dighified surroundings. Children eyed little-known 
cousins vith distirust and they, together with (12)-up > ■ 
da.ughters, had to (13) th^ company (14) piano-glaying or ^ 
recitations while 'their own parents beamed complacently! • ( 
There was the barbed ex&hange of news, gossip, gloomy 
prdgnostications about absent relatives , 'reminiscences of 
Uncle Albert and sha;:ed (15) of ■ childhood.^ The mask of, 
formality gradually slipped and the ^natural "individual , ^ 
self-satisfied or envious, chairit^l^ or spiteful, assumed 
control. ' , ' " ^ f ~ 
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